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Abstract 

This paper discusses an exploratory study to investigate the existence, and nature, of student 
problem formulation and resolution processes in an undergraduate online Problem-Based 
Learning (PBL) course in Agricultural Sciences. We report on the use of a content analysis 
instrument developed to measure problem formulation and resolution (PFR) processes in online 
asynchronous discussions (Murphy, 2004a, 2004b) to analyze students' text-based, online 
discussions. The results offer evidence that students do engage in problem formulation and 
resolution and that these processes appear to be consistent with the PBL process carried out in 
this course. However, the nature of the PBL pedagogy, at least in this instructional context, ties 
the PBL problems to be solved tightly to a marked assignment structure and, therefore, appears to 
restrict the PFR process in its early and late stages. 

Keywords: online learning; problem-based learning; problem-solving; Constructivism; 
instructional design; content analysis 


Introduction 

In a discussion on the International Forum of Educational Technology and Society, Nichols and 
Anderson (2005, 'll 12) make two important points about instructional design for e-Learning: 

1. E-Learning pedagogies must be defensible, used with some reference to proven 
educational practice and underpinned by accepted educational theory. 

2. E-Learning pedagogies are evolving in the sense that new modes of practice and 
enhanced technological tools are continually emerging. E-Learning practice cannot 
remain static, but should instead seek to make the most of new opportunities. 

In essence, in designing e-Learning, instructional designers must be guided by research and 
theory and must be willing to use it to guide them to new and justified instructional practices. In 
this paper, we examine the use in e-Learning of an established and well-researched pedagogy. 
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Problem-Based Learning (PBL), an educational strategy in which complex, ill-structured 
problems serve as the context and the stimulus for learning and then we report on a study to 
investigate the existence and nature of student problem formulation and resolution processes in an 
undergraduate online Problem-Based Learning (PBL) course in Agricultural Sciences. 

Literature Review 

PBL contrasts with more traditional subject based approaches where students are first taught a 
body of knowledge and then may have an opportunity to apply what they have learned to sample 
problems. With PBL, students work collaboratively in groups to identify what they need to learn 
in order to understand the problem and to learn about the broader concepts and principles related 
to the problem. PBL, therefore, is designed to encourage active participation by the students by 
immersing them in a situation. It requires them to define their own learning needs within broad 
goals set by faculty, then to identify and search for the knowledge that they need to obtain in 
order to solve the problem. 

PBL, as a pedagogical approach, was developed the 1960s and has been most widely used in 
Medical Education. It has also been employed in a range of other fields, however, including 
Nursing, Dentistry and Agriculture (Barrows, 1996, 1998; Boud & Faletti, 1991; Savery & Duffy, 
2001 ). 


Defining PBL 

Barrows (1998) articulated what has become one of the most widely used definitions of PBL. He 
termed it “authentic PBL” and argued that it has four key characteristics: 

1 . Problem-based. It begins with the presentation of a real life (authentic) problem stated as 
it might be encountered by practitioners. 

2. Problem-solving. It supports the application of problem-solving skills required in 
“clinical practice.” The role of the instructor is to facilitate the application and 
development of effective problem-solving processes. 

3. Student-centred. Students assume responsibility for their own learning and faculty act as 
facilitators. Instructors must avoid making students dependent on them for what they 
should learn and know. 

4. Self-directed learning. It develops research skills. Students need to learn how to get 
information when it is needed and will be current, as this is an essential skill for 
professional performance. 

5. Reflection. This should take place following the completion of problem work, preferably 
through group discussion, and is meant to enhance transfer of learning to new problems. 

Research on PBL 

Research on PBL, especially as used in medical schools, has focused primarily on comparing the 
outcomes of PBL methods to more traditional instruction (Albanese, 2000; Albanese & Mitchell, 
1993; Colliver, 2000; Smits, Verbeek & Buisonje, 2002; Vernon & Blake, 1993). Much of this 
research has focused on the effectiveness of the pedagogy to foster learning. 
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A review of the literature on effectiveness on PBL in face-to-face instructional settings leads to 
mixed conclusions. Vernon and Blake (1993) used meta-analysis to compare 35 studies of PBL in 
medical education. The authors found that PBL was superior with respect to students’ clinical 
performance, but PBL and traditional methods did not differ substantially on tests of factual 
knowledge. Albanese and Mitchell (1993) produced similar findings. Students of conventional 
curricula outperformed PBL students on measures of basic science while PBL students scored 
higher on clinical examinations. 

A more recent study (Dochy, Segers, Van den Bossche & Gijbels, 2003) produced similar overall 
results. They found a mild negative effect favouring traditional approaches for the assessment of 
student knowledge, although these differences were encountered in first and second year of 
medical school and evened out in the last two years. PBL students gained slightly less knowledge 
but remembered more of it over time (retention). The results for skills were consistently positive 
favouring the PBL curriculum. 

Less work has been done on the specific learning processes occurring in students engaged in PBL 
(Arts, Gijselaers & Segers, 2002; Hmelo, Gotterer & Bransford, 1997; Kamin, O’Sullivan & 
Deterding, 2001; Norman & Schmidt, 1992). Gijbels, Dochy, Van den Bossche, & Segers (2005) 
evaluated 40 studies in order to examine the depth of student knowledge acquisition. They 
applied Sugrue’s (1995) integrated model of the cognitive components of problem-solving, which 
proposes that learners’ knowledge structures consist of three levels: 1) understanding of concepts; 
2) understanding of the principles linking concepts; and 3) understanding the links from concepts 
and principles to conditions and procedures for application. The results supported PBL at all three 
levels but showed that it had the most positive effects when constructs were being assessed at the 
level of understanding principles that link concepts. 

Research on the applicability of this approach in an online, Distance Education, context is also 
limited (e.g., Atan, Sulaiman & Idrus, 2005; Brown, Johnson, Lima, Boyer, Butler, et ah, 2004; 
Chanlin & Chan, 2004; Ortiz, 2004), although there has been some more extensive work on 
blended learning or distributed problem-based learning (dPBL) (e.g., Barrows, 2002; Bjorck, 
2002; Bowdish, Chauvin, Kreisman & Britt, 2003; Cheaney & Ingebritsen, 2005; Lehtinen, 2002; 
Lopez-Ortiz & Lin, 2005; Lou, 2004; Orrill, 2002; Pearson, 2006; Ronteltap & Eurelings; 2002). 

Arts, Gijselaers, and Segers (2002) reported the redesign of a course in Business Education to 
offer PBL in a blended learning environment. Students accessed problem materials on CD-Rom 
and on the Internet, but met in face-to-face PBL groups. Scores on a knowledge application test 
indicated that the redesigned PBL-format contributed significantly to improved cognitive gains 
compared to the regular PBL-setting. However, this was not a fully online PBL course. 

Brown et al. (2004) discuss a problem-based learning simulation delivered via the Web for 
middle and high school students during a five-week period. Both males and females significantly 
increased their knowledge scores after the completion of the simulation, but from the project 
description, it seems clear that this instruction did not meet Barrows’ criteria for authentic PBL. 

Atan, Sulaiman, and Idrus (2005) compared the performances of students in an undergraduate 
Physics lesson using the Web-based PBL to that of the same students using a Web-based 
Content-Based Learning (CBL) in a subsequent lesson. Results significantly favoured the Web- 
based PBL approach, but are based on a brief, 7-item posttest. The PBL treatment was clearly 
quite limited in scope and likely could not be considered authentic PBL. 

Finally, Pearson (2006) described the design, implementation and evaluation of a module in 
Business Education in which PBL was used to investigate the challenges associated with the 
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adoption and use of ICT in Hong Kong secondary school classrooms. An evaluation examined 
five questions dealing with the implementation of PBL, the extent to which PBL facilitated 
academic discourse, the extent of new knowledge about ICT created, the role of the tutor, and the 
online learning environment provided, but did not assess the instructional effectiveness of the 
approach. 

Overall, then, there appears to be some evidence that PBL is an effective pedagogy when used 
over time in whole curricula. However, given the mixed results, it is uncertain that it would make 
any difference in instruction of shorter duration and it is not yet clear if it can be effectively 
employed in an online context. 

Nonetheless, experimental research studies and quantitative review methods may permit 
relatively strong statements of certainty about effectiveness, but these statements are typically 
quite broad - e.g., PBL facilitates the learning of clinical reasoning skills. Such conclusions still 
tell us little about the cognitive processes underlying learning in such contexts and how specific 
instructional strategies affect such processes. For instance, Barrows and other proponents of PBL 
have argued strongly that this instructional approach sets the conditions for effective and deep 
learning of both disciplinary knowledge and of problem-solving (e.g., Albanese, 2000; Barrows, 
1998, Norman & Schmidt, 1992, 2000). Moreover, Barrows (1998) claimed that only “authentic” 
PBL could foster both the acquisition of a deeply understood knowledge integrated from a variety 
of disciplines and the development of effective clinical problem-solving [emphasis added]. 

Purpose of the Study 

While all the characteristics of PBL can be seen as important, problem-solving may be key. What 
does it mean to support problem-solving skills required in clinical or professional practice? What 
exactly are these skills? How does the instructor facilitate the application and development of 
effective problem-solving processes and how would one know that problem-solving was 
occurring? 

This paper reports on an exploratory study to investigate the existence and nature of student 
problem formulation and resolution processes in an undergraduate online Problem-Based 
Learning (PBL) course in Agricultural Sciences. We accept Jonassen’s (1997) model for solving 
ill-structured problems, which holds that problem-solving consists of two main categories: 
Problem Formulation and Problem Resolution (the PFR process). We describe the use of a 
content analysis instrument developed to measure problem formulation and resolution processes 
in online asynchronous discussions (Muiphy, 2004a, 2004b) to analyze students’ text-based, 
online discussions and the modification of this instrument to more closely match the specific 
problem-solving processes occurring in PBL. The following research questions guided the 
investigation: 

1. What evidence is there that undergraduate Agricultural Sciences students 

exhibit problem-solving behaviours and skills in an online PBL course? 

2. What is the nature of the problem-solving process that students apply when 

engaged in online PBL activities? 
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Methodology 


Participants 

The participants in this study consisted of the 11 students registered in the course and their 
instructor. The students were divided into two PBL groups of five and six students respectively 
and one student dropped the course part way through Case 1 (early October). The membership of 
the two groups was restructured after each case so that all students in the course worked with 
each other at some point in the course. 

Research Setting 

This study examined student and instructor interactions in an online course on AgroEcology, one 
of two online PBL courses taught in the Faculty of Land and Food Systems at a Western 
Canadian university. These courses were delivered using WebCT™ Campus Edition 3.8. 

The use of incomplete case studies. Barrows (1998) states that a PBL approach must be 
problem-based - i.e., it should begin with the presentation of an ill-defined, complex, authentic 
problem. These problems usually consist of descriptions of sets of events that need explanation 
and provide only limited information. The course material in AgroEcology was introduced 
through the study of four cases impacting on the practice of Agroecology: 1) grazing ecosystems; 
2) organic vegetable production; 3) tree fruit agroecosystems; and 4) genetically modified 
organisms and rural communities. Students were asked to play the role of consultants to “clients” 
presented in the case and the four assignments (one for each case) were structured as reports to 
these clients. 

Each case was comprised of multiple rounds, each of which includes several disclosures. These 
disclosures presented students with the scenario that introduces the problem that they are being 
asked to address (See Figure 1) or else provides more information about it (supplementary 
disclosures). In most cases, supplementary disclosures are made available as learners discuss the 
scenario and identify further information that they require. 
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Figure 1 . AgroEcology Case 2 problem statement. 


«#RO 240 C«M 2 R*«M • 



Introduction and Problem Statement 


The BC Association for Regenerative AgncUtue has recent)/ employed you as site 
inspectors to certify organic vegetable producers in the Fraser valley 

On your first day at wot*, you discover that your initial task involves an inspection o< 
Fnesen Farms in Alder grove. BC to ensure that their techniques and practices meet 
the gudetnes presented by the certifying bod/ (BC Association for Regenerative 
Agneuture) 

Note: As part o< your responsibikties you are also obliged to assist organic growers by 
answenng their production-related questions and helping them to address their goals to 
manage sustainable agroecosystems 

You ultimate assignment at the end of the case wl be to prepare a report that 
addresses the following issues 

• a decision on re-certification that veil be forwarded to the BC Associadon for 
Regenerative Agneuture. including the rationale for you decision 

• comment and proposals for appropriate' crop rotations for Fnesen Farms which 
enhance soil fertility, and reduce the incidence of pests and disease 

• identification of the general ecoio^cai pnnciptes that are applicable on the farm 
and how they relate to mana^ng a sustainable agroecosystem 


In AgroEcology, each PBL group had two available discussion forums: a Process and Evaluation 
Forum (See Figure 2) used by the group to review and discuss ground rules for collaboration as 
well as the overall process for conducting work within each working round, and a Working 
(Discussion) Forum used by the group to cany out the actual PBL process itself. 
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Figure 2. AgroEcology discussion groups. 



The Working Forum took the place of face-to-face meetings in which learners engage in various 
group processes including definition of the problem, development of working hypotheses, 
organization of the elements of the problem, agreement on research tasks and reporting back on 
research completed. The instructor participated by monitoring the discussions and making timely 
postings to encourage student participation, guiding the discussion of controversial points, 
ensuring that concepts were mastered, encouraging depth of thinking, and verifying the quality of 
resources used. 


Analysis 

Garrison, Cleveland-Innes, Koole, and Kappleman (2006) argue that a sound theoretical 
framework and model is essential to address validity and to guide a transcript analysis. It is the 
research question and purpose of the discourse that should determine the model and coding 
scheme used. Several transcript analysis instruments have been developed recently to measure 
critical thinking in an online environment (e.g., Garrison, Anderson, & Archer, 2001; McLean, 
2005; Meyer, 2004). While they appear to be closely related processes however, it is not certain 
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that problem-solving and critical thinking are the same thing. A common definition of critical 
thinking (e.g., Bullen, 1998; Bailin, Case, Coombs & Daniels, 1999) is that it is thinking that is 
reasonable and reflective and focused on what to believe or do. Garrison et al. (2001, p. 8) view 
the outcome of critical thinking to be the acquisition of deep and meaningful understanding and 
to include problem-solving. Bailin et al. (1999), on the other hand, consider problem-solving as 
an arena in which critical thinking may take place. Jonassen (2000), however, regards problem- 
solving as a more distinct process. A problem is an unknown entity in some situations (the 
difference between a goal state and a current state) and problem-solving is the process of finding 
this unknown (p. 65). 

This analysis was carried out using a content analysis instrument recently developed to measure 
Problem Formulation and Resolution (PFR) processes in online asynchronous discussions 
(Murphy, 2004a, 2004b) because it was designed specifically to measure the problem-solving 
process which was the focus of the research questions in this study. This instrument is based on 
the conceptual framework of Jonassen’s (1997) model for solving ill-structured problems and 
consists of two main categories: Problem Formulation and Problem Resolution (the PFR process). 
Each category is in turn divided into several sub-processes and a series of 19 indicators for these 
processes (See Table 1). 

In a previous pilot study (Kenny & Bullen, 2005), we conducted a post-hoc, descriptive content 
analysis of all discussions of the Working Forums for both PBL groups for Case 1. This consisted 
of 348 separate postings for Group 1 and 309 postings for Group 2. This initial analysis applied 
Murphy’s (2004b) revised instrument. On the basis on this pilot analysis, we then further 
modified the instrument to more closely match the PBL process occurring in this online course. 
Murphy (2004b) points out that, to accurately measure the construct (problem-solving) that they 
purport to measure, such instruments must adequately encompass important aspects of the 
construct and eliminate aspects distinct from, or surplus to, the intended construct being 
measured. To do so, the construct can be refined empirically through its manifestations in real 
contexts. The version of the instrument used in this study is shown in Table 1. 

Table 1 . Processes and Indicators for Identifying and Measuring PFR in PBL. 


Process 

Indicator 

Problem formulation 

Articulating problem space 

No indicator provided 

Defining problem space 

Agreeing with problem as presented in OAD 
Specifying ways that the problem manifests itself 
Redefining problem within problem space 
Minimizing and or denying problem 
Identifying extent of problem 
Identifying causes of problem 
Articulating a problem outside problem space 

Building knowledge 

Identifying unknowns in knowledge 
Seeking information to resolve lack ofknowiedge 
Clarifying (meaning, importance, accuracy of) 
information 

Accessing and reporting on sources of information 
Identifying value of information 
Reflecting on one’s thinking 

Problem resolution 

Identifying solutions 

Proposing solutions 
Hvpothesizing about solutions 

Evaluating solutions 

Agreeing with solutions proposed by others 
Weighing and comparing alternative solutions 
Critiquing solutions 

Rejecting eliminating solutions judged unworkable 

Reaching conclusions 

Coming to agreement about solutions 

Acting on solutions 

Planning to take action to resolve the problem 
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Specifically, we discovered that several aspects of the instrument either did not apply in PBL as 
implemented in this particular course, or else we found the description of some processes and 
indicators confusing, missing or out of order. First, no indicator was provided for the process, 
articulating problem space. This process is a part of the overall PFR process, but, as was also the 
case in Murphy’s studies, PBL problems are given to the learners, so it is not necessary to include 
indicators in the instrument. 

Second, we added a new indicator, Clarifying Information, because our analysis revealed that 
students spent a considerable part of their discussion clarifying the meaning of, the importance of, 
or the accuracy or currency of information. This indicator added specificity to the process of 
building knowledge. An example of a posting from Case 1 demonstrating this process is listed 
below: 


At the end, AUM does mean the number of animals, but this is where the unit kg 
comes from. 

1 hope this is clear to everyone. Ask me more questions if anything doesn 't make 
sense ( it's 1:39am and i'm not thinking too clearly anymore . . .). 

Third, we added a new process, Reaching Conclusions, to the second phase of Problem 
Resolution. In Murphy’s (2004b) most recent version of the instrument, this was included as an 
indicator under the process, Acting on Solutions. We considered this to be a process in and of 
itself and that it followed directly from the process of evaluating solutions, i.e., one which takes 
place prior to actually taking action to implement a solution. Murphy (2004b) added this category 
as a result of the analysis leading to the revisions of the first version of the instrument. She noted 
that “there were cases where participants indicated intentions, but did not explicitly state a plan of 
action. Instead, they may have simply reached a conclusion . . . (p. 350).” 

Finally, we kept the last process, Acting on Solutions. In her first paper, Murphy (2004a) notes 
that Acting on Solutions represents the culmination of PFR “whereby individuals can apply the 
results of a problem in an actual context (p. 10).” Since we had moved Reaching Conclusions up 
to become a new process, this left only one indicator for this process, which was “planning to 
take action to resolve the problem.” For instance, Murphy (2004a, p. 12) gives the following 
example of Planning to Act: “Personally, I have decided to speak English the first day of classes.” 
This is clearly a statement about what the individual will do as a result of the problem solving 
process. 

However, we concluded that this stage was not relevant for the PBL process as used in this course 
because students were not actually asked to go out on the farm and apply the solutions. The 
solution was, in effect, the final assignment for the case, the mock consultant’s report. Initially, 
the first author considered the following posting from Case 1 an example of planning to act if this 
process included creating the report: 

I was hoping we'd all "vote" for one, although i know it's early, but I think Joan 
will need some cow math for tomorrow, so for now I am gonna go with my plan 
as described above and do herd calculations. It's not final, but just to get some 
numbers . . . :) 

After discussion, we decided it did not fit into Planning to Act because it doesn't refer to how the 
student might apply a solution they worked out. Instead, it simply states what more he is going to 
do to reach a solution and refers to the organization of the PBL group activities in order to move 
forward with the class process. We eventually included postings of this sort into a catch all 
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organizational category that we labelled PBL Organization. This may well be an issue of the 
“artificiality” of PBL problems in that they are realistic, but not actual real life activities. 

Garrison et al. (2006) characterize transcript analysis as an exploratory, qualitative methodology 
and point out that the goal is descriptive, to attempt to understand the existing interactions. As 
such, they recommend the use of a negotiated agreement in which coders first code transcripts 
separately and then discuss their respective coding to arrive at a final version in which coded 
messages have been brought into alignment. This approach was applied to this analysis. The first 
author coded the discussions for both Group 1 Case 2 and for Group 2 Case 2, while a graduate 
assistant (the third author) also coded Group 1 Case 2 and the second author coded the Group 2 
Case 2 discussion. To code, we followed the data analysis processes outlined in Murphy and 
Ciszewska-CaiT (2005), in which the authors advocate the paragraph as unit of analysis and 
recommend a three level analysis process: 1) first to code units at the level of the category (i.e., is 
this unit an example problem formulation/understanding or is it problem solving/ resolution?); 2) 
next to re-code at the level of the process; and 3) to code a third time using the indicators. While 
the paragraph was taken as the unit of analysis, we coded each paragraph for instances of (one 
instance each of) multiple indicators. Finally, each pair of coders met in several sessions to 
discuss and reconcile disagreements and to arrive at a negotiated agreement as reported below. 

Results 

As indicated above, we followed a three level analysis process: 1) to first code paragraphs at the 
level of the category, i.e., as problem formulation or as problem resolution; 2) to next re-code at 
the level of the process; and 3) to code a third time using the indicators. The total number of 
postings for the Case 2 Group 1 discussion was 237 and 230 for Group 2. Table 2 shows the 
results of the coding at the level of category for Case 2 Groups 1 and 2. Nearly half of the 
postings for Group 1 were viewed as problem formulation while nearly two thirds of the postings 
for Group 2 were placed in that category, while postings judged as problem resolution ranged 
around twenty percent. One third of all postings for Group 1 and 20 percent for Group 2 were 
judged as organizational or social and not representative of the problem-solving process. 


Table 2. Coding of PFR Categories in AgroEcology Case 2 - First Pass 


Category 

P Codings - 
Group 1 

% of 
Codings* 

p Codings - 
Group 2 

% of Codings 

Problem Formulation 

228 

48.7 

365 

633 

Problem Resolution 

85 

182 

147 

/< 

PBL Organization 

144 

30.8 

60 

19.5 

Social Postings 

11 

2.4 

5 

1.5 


a. Pacsnlags inTafcls 2 is bassd on ths total number of codings made overall during the first pass through 
the data. This includes PBL Organization and Social po6 tings. 


Table 3 provides the results of the codings at the level of process for Case 2 Groups 1 and 2. Two 
thirds of the postings for both Group 1 and Group 2 were judged to involve the process of 
building knowledge. Within the category of Problem Formulation, the process of defining 
problem space was represented by only five percent of the postings for each group. 
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Table 3. Coding of PFR Processes in AgroEcology Case 2 Group 2- Second Pass. 


Process 

# Codings 
-Group 1 

% of 
C odings 6 

# Codings - 
Group 2 

% of Codings 

Problem Formulation 

Defining problem space 

17 

5.1 

25 

4.8 

Building knowledge 

217 

65.6 

349 

66.3 

Problem Resolution 

IdentifSing solutions 

40 

12.1 

96 

13.1 

Evaluating solutions 

41 

12.4 

44 

8.4 

Reaching conclusions 

16 

4.8 

12 

23 

Acting on solutions 

0 

0.0 

0 

0.0 


b. Percentage in Table 3 is basec on the total number of Process level codtnss mace during the second pass 


through the data. PBL Organization and Social postings tv ere not included. 


Less than one third of the postings for each involved the category of Problem Resolution and 
these were fairly evenly divided between identifying and evaluating solutions. The two groups 
engaged in the process of reaching conclusions in less than five percent of their activities and 
there were no coded instances of acting on conclusions. 

Table 4 provides the results of the codings at the level of indicator for Case 2 Groups 1 and 2. 
These findings necessarily mirror those of Pass 2 and provide detail about the nature of the 
problem-solving processes. Five percent of codings fell within the process of Defining the 
Problem Space. Of these, the majority (four percent of the total indicator codings) were assessed 
as identifying the extent of the problem. The majority of codings in Pass 2 were judged as being 
representative of the process of Building Knowledge. 
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Table 4. Coding of PFR Processes in AgroEcology Case 2 Group 2- Third Pass. 


Indicator 

# Codings - 
Group 1 

% of 
Codings 1 

- Codings - 
Group 2 

% of 
Codings 

Defining problem space 

Agreeing with problem 

1 

0.3 

0 

0 

Ways problem manifests 

1 

0.3 

7 

1.1 

Redefining problem 

3 

0.8 

0 

0 

Minimizing problem 

0 

0.0 

0 

0 

Extent of problem 

14 

4.0 

13 

2.1 

Cause of problem 

0 

0.0 

7 

1.1 

Outside problem space 

0 

0.0 

0 

0 

Building kncnvledge 

Identifving unknowns 

23 

6.0 

41 

6.7 

Seeking information 

37 

9.3 

57 

93 

Clarifying information 

70 

17.5 

70 

11.4 

Reporting infoimation 

97 

243 

222 

362 

Value of information 

26 

6.5 

13 

2.1 

Reflecting on thinking 

10 

2.5 

2 

03 

Identifying solutions 

Proposing solutions 

35 

8.8 

73 

11.9 

Hypothesizing 

22 

5.5 

65 

10.6 

E\uiuating solutions 

Agreeing with solutions 

19 

4.8 

13 

2.1 

Weighing alternatives 

15 

3.8 

10 

1.6 

Critiquing solutions 

5 

1.3 

16 

2.6 

Rejecting solutions 

5 

1.3 

5 

0.8 

Reaching conclusions 

Caning to agreement 

16 

4.0 

0 

0.0 

Acting on solutions 

Ranting to take action 

0 

0.0 

0 

0.0 


c. Psrcsntase in Table 4 is basec on die total numb er of Indicator level codings made during the third pass 


through the data. ?BL Organization and Social postings were not included. 


Most of these were seen as examples of mainly three indicators: 

1. Accessing and reporting sources of information, which encompassed nearly twenty- 
five percent of the indicator codings for Group 1 and well over one third for Group 2. 

2. Clarifying the meaning, importance or accuracy of information, which covered nearly 
eighteen percent of postings at this level for Group 1, and over ten percent for Group 2. 

3. Seeking information, which involved nearly ten percent of the activities of both 
groups. 

The processes of Identifying Solutions and Evaluating Solutions were the next most highly 
represented processes in Pass 2 and evenly represented. The process of Identifying Solutions was 
described by two indicators, proposing solutions and hypothesizing about these solutions. Both 
indicators were relatively evenly covered in this analysis and ranged from nearly six percent to 
twelve percent of the codings in Pass 3. The process of Evaluating Solutions included four 
indicators: agreeing with solutions proposed by others, weighing and comparing solutions, 
critiquing solutions, and rejecting solutions judged unworkable. The coverage of these indicators 
in our analysis was quite variable, ranging from a high of nearly five percent for agreeing with 
solutions for Group 1 to a low of less than one percent for Group 2. 

The process of Reaching Conclusions was described by only one indicator, coming to agreement 
about solutions. This indicator was judged as occurring in four percent of the codings for Group 
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1, but the coders for Group 2 were unable to agree if the 12 instances of this process found in 
Pass 2 were described by this indicator. Rather, it was felt that a new indicator was needed. 

Discussion 


The PFR Process and PBL 

The students in this course were engaged in a highly structured PBL process. In each of the four 
cases, they were presented with an explicit, if relatively ill-defined, problem situation to resolve 
and were asked to produce a solution in a specific format. For example, for Case 2, they were 
asked to produce a consultant’s report in the form of recommendation of the re-certification of a 
farm as organic and to provide a crop rotation plan (see Figure 1). Since this solution was also a 
course assignment (worth marks), they were not likely to deviate substantially from it. The 
overall results show that all aspects of the problem formulation and resolution process were being 
fostered within the parameters of the PBL process applied in this course with the exception of the 
process of Acting on Solutions. 

As indicated previously, analysis of the PFR process occurred in three stages (passes): 1) first at 
the level of category (Problem Formulation and Problem Resolution); 2) at the level of sub 
processes for each category (see Table 1); and 3) at the level of indicators for each process. 
Nearly half of the postings for Group 1 and two thirds of the postings for Group 2 were viewed as 
Problem Formulation, while postings judged as Problem Resolution ranged around twenty 
percent. The PBL process is structured to direct learners, within their collaborative groups, to 
quickly determine what they do and do not know, then to conduct research to fill in the missing 
information and report back to the group. Only then do they attempt to come to conclusions or 
develop solutions. Therefore, where students are new to the subject domain, it is not unexpected 
that a substantial part of the activity of the group be focused on Problem Formulation, especially 
on the process of building knowledge. This appeared to be the case in this course since the course 
was offered at the second year undergraduate level (Agro 260) and the course instructor noted 
that these students were just getting used to the PBL model of instruction (K. Nolan, personal 
communication, July 15, 2005). 

The analysis of the PFR processes occurred in the second stage (Pass 2). No indicator was 
provided for the process of Articulating the Problem Space and it was not coded. As in Muiphy’s 
(2004a) study, while this process is recognized as a part of the PFR model, explicit and relatively 
detailed problems were given to the students and they were not required to engage in this activity. 
Moreover, for both PBL groups, the investigators found low activity (five percent) in the first 
category of defining the problem space. This is not surprising given the nature of the PBL process 
and the strong structure of this course. Since the students were required to produce written 
assignments for marks based on the problems as presented, they were unlikely to disagree with it 
as stated or to attempt to redefine it. Indeed, for both PBL groups, the greatest number of postings 
coded as representing this process fell under the indicator, Identifying the Extent of the Problem, 
which would indicate that the students were not engaged in redefining the problem, articulating 
new problems, or otherwise redefining the problem space. In PBL as represented in this course, 
the problem space comes pre-defined. 

For each group, the investigators discovered the highest activity by a wide margin under the 
process of Building Knowledge. Whether conducted in a face-to-face instructional setting or 
online, the PBL process puts considerable onus on the students to decide what they do and do not 
know about the problem to be solved and then to conduct research on those topics which are 
unknown. Topics to be researched are typically divided up between group members, who then 
report back to the group with their findings. In Case 2 of this course at least, the students clearly 


Kenny, Bullen & Loftus- Problem Formulation and Resolution in Online Problem-Based Learning 


14 


focused mainly on seeking and reporting back information and on clarifying the meaning and the 
importance of what they had found. 

After Building Knowledge, the next mostly highly coded processes were those of Identifying 
Solutions and Evaluating Solutions. Students in this course engaged in both processes nearly 
equally. The solutions to the problems the students were required to solve in this course, as in all 
PBL instruction, were complex and composed of multiple components. Group members needed 
to determine the nature of these sub-solutions and to agree on them. It was, therefore, to be 
expected that there would be evidence of the processes of identifying solutions, evaluating them 
and drawing conclusions. While they differed in the quantity of their assessments, both sets of 
coders found multiple instances of most indicators of these processes. The one exception was 
that, while they found 12 instances of the process, Reaching Conclusions, the investigators coding 
Group 2 did not code any instances of the indicator, Coming to Conclusions. Rather, after 
extensive discussion, we decided that a new indicator, Proposing Conclusions, was needed. The 
following postings from Group 2 members represent this indicator: 

I agree . . . If any certification is granted, there are a lot of things that need to be 
substantiated. It is strictly against the guidelines to use animal manure from 
animals that are not 'organic' ones. Because this hasn ’t been specified, I think it 
is safe to grant the farm a conditional certification as Joan has suggested. 


And 


SO as we have been discussing in the thread below on water quality, maybe the 
pH level in the water (resulting in it not being classified as acceptable water to 
rinse the veggies in) is reason enough for Friesen farms not to be recertified? 

What do you all think? 

The most significant finding in this second analysis was that neither investigator coded any 
instances of Acting on Solutions. This is consistent with our concern about whether this stage is 
relevant for the PBL process because students were not actually asked to go out on a farm to 
actually apply the solutions. Rather, they had to write up their solutions as recommendations in a 
course assignment. Since these assignments were not available to the investigators, there was not 
a visible product to allow us to verify the existence of this process. This finding is also consistent 
with the nature of the problems that students were given in this case. That is, they did not require 
solutions to be implemented but only that the solutions are articulated in a “consultant’s” report. 
One would not reasonably expect to find examples of Acting on Solutions given this type of 
assignment. 


Limitations of the Study 

There are two potentially related problems with the use of the PFR instrument as applied in our 
analysis: 1) the choice of the unit of analysis; and 2) the accuracy of the current instrument for 
representing the PFR processes in a PBL context. A third potential limitation relates to the 
possibility that the transcripts analysed may only be reflecting a restricted component of the PBL 
process. 

Choice of the unit of analysis. As indicated above, we assumed the paragraph to be the unit of 
analysis. This procedure allowed us to code the same postings consistently among coders. Fahy 
(2001) pointed out that when the focus is on the meaning [original emphasis] of the interaction of 
the conference, the unit of analysis [original emphasis] must be something obvious and constant 
within transcripts. He concluded that this should be the sentence or independent clauses that 
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could be structured as sentences if punctuated differently. The semantic or notional meaning may 
indeed transcend textual structures, but structural elements of text help form and convey [original 
emphasis] the notional relationships of the argument (Fahy, 2001, fl2). Murphy {personal 
communication, October 4, 2005), however, disagrees with the choice of the sentence as the unit 
of analysis, arguing that it is insufficient to convey meaning and represents a potentially onerous 
analysis process. Flaving now completed two full analyses of PBL cases from this course, we 
would agree with Murphy that the sentence is rarely sufficient to convey full meaning in this 
circumstance. We also found, however, that it was often difficult to determine exactly when a part 
of a posting was a paragraph (e.g., many times, one sentence was separated from the others) and, 
even when the paragraph structure was clear, this unit was often still insufficient to convey 
meaning. We found many instances where the meaning of a posting carried over two or more 
paragraphs. For instance, on a number of occasions, the second author argued that the two 
indicators, Proposing Solutions and Hypothesizing about Solutions, were inseparable and should 
be one. Yet, all three coders found instances where a student proposed a solution in one paragraph 
and then explained it in the following paragraph. 

Representing the PFR processes in a PBL context. A second issue concerns the suitability 
of the PFR instrument for measuring problem solving in a PBL context. Murphy and Ciszewska- 
Carr (2005) obtained a high level of inter-coder agreement (a kappa coefficient of 0.825 for the 
two categories of Problem Formulation and Problem Resolution), but they also used the 
instrument to analyze a discussion that was specifically structured to parallel the problem 
formulation and resolution process. The discussion was divided into eight tasks, each of which 
asked the learners to apply a step in the PFR model; e.g., the first task required participants to 
reflect on their initial knowledge of the problem and to post a message describing their 
understanding of the problem. 

Our results indicate that the PBL process can be seen to broadly follow the PFR scheme. The 
process of Defining the Problem, however, is minimally represented and only then by one 
principle indicator (Identifying the Extent of the Problem) and there were no instances of the 
process, Acting on Solutions. The degree to which learners fully engage in the PBL process 
depends on guidance they receive via instructional materials and the interventions of the PBL 
instructor, their understanding of the process and the complexity of the material being engaged. In 
essence, PBL represents a real life problem-solving activity, but one which may not cleanly 
compliment the PFR model. 

This being said, as discussed above, we did modify the original instrument to more fully match 
the PBL process on the basis of findings of our pilot study, (Kenny & Bullen, 2005), so one 
would expect better agreement. It may be necessary to modify the instrument further or else select 
a different instrument. In particular, the terminology of category, Problem Formulation, creates 
confusion because, on the face, it suggests that students would be engaged in defining what the 
problem is. The PBL approach, however, as manifested in this course, provides students with the 
problem and, through supplementary disclosures, much of the additional relevant information. 
Their task is to clearly identify the nature of the problem, identify what missing information they 
need, and to develop solutions. There was no need for the learners to formulate the problem in the 
sense represented by many of the PFR indicators, such as “specifying ways that the problem 
manifests itself’ or “redefining problem within problem space.” 

Rather than a question of clarifying terminology, this may require the recognition of the overlap 
in these problem-solving processes, that is, to accept that problem-solving is highly recursive in 
nature. Muiphy (2004a) clearly views Problem Formulation as both understanding the problem 
within its context and building a body of knowledge about the problem area. In applying the 
instrument in this analysis, we frequently found that those postings we regarded as Building 
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Knowledge fit more within the realm of Problem Resolution than Problem Formulation because 
they focused on gathering and clarifying information for constructing solutions rather than 
clarifying the nature of the problem itself. An example follows of a posting which might fit into 
either (or both) the problem formulation and problem resolution categories. The posting discusses 
the use of commercial and “home grown” manure on an organic farm. 

Right now, I’m not sure what 'words of wisdom' I can offer Ann [the fanner in 
the case study for whom the report if being written] (I myself am still unsure 
about integration of all the research info into specific applications for the case). 

The only thing I can suggest from this info is that Ann is safer to use processed 
manure from a company, although this is likely to be more expensive than 
unprocessed manure from a producer. If economics are a concern, and Ann 
wishes to continue using manure from a producer (or from her own animals; this 
issue is still up in the air) she must ensure that it ages long enough to avoid the 
risk of pathogens and introduction of weeds. I think once we have a firm grasp on 
the soil condition, we can make a good decision on what type of fertilizer should 
be used ( i.e. from what animal and amount it has been processed) and whether or 
not an imals should be kept on the farm. 

Clearly, when the students refer to integrating research into specific applications for the case, they 
are engaging in developing a solution and, therefore, Problem Resolution. Learning more about 
the issue of which form of manure to use, however, could be seen as either understanding the 
problem or elucidating a solution. 

Possible use of other methods of communication in the PBL process. Did the students also use 
email or instant messaging, talk on the telephone, or meet in person? Did our analysis miss a 
significant part of the PBL group process, and hence, the problem-solving activity that occurred 
during the course? While we did not examine this question directly, there are several reasons to 
believe that the students in this course did not use other methods to collaborate and that they 
restricted their PBL deliberations to the working forum provided. First, the course instructions, 
which directed the students to use the discussion forum provided for group collaborations, were 
clear and very detailed. Moreover, the instructor was present from the start and very active in 
guiding the discussions throughout and she only used the discussion forum. The PBL groups 
appeared to closely follow her guidance in the PBL process and to restrict themselves to the 
working forum in particular (i.e., there was little use even of the Process and Evaluation Forum). 
Second, we found no instance in any of the transcripts of students asking fellow members of the 
PBL group to use a different communications medium (e.g., exchanging instant messaging 
usernames). In fact, in several postings, group members asked if another student was currently 
online, that is, the PBL group members were sometimes using the discussion forum as a form of 
instant messaging by posting and then waiting for a reply. And finally, we did find one instance 
where two of the PBL group members discussed encountering each other in a different, face-to- 
face, course. They mentioned how nice it was to meet in person, but made no reference to 
meeting in this way for the PBL process. In fact, one of the students in one group moved to 
Central America during the course and communicated from that location for half the course, so 
meeting in person was not in any case always feasible. 

Conclusions and Implications 

This was an exploratory study designed to examine two questions within the context of the 
specific, online PBL course examined: 1) what evidence is there that PBL fosters problem- 
solving behaviours and skills; and 2) what is the nature of the problem-solving process which 
students apply when engaged in PBL activities? From our analysis, it appears that online PBL can 
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foster problem-solving behaviours in learners, at least in the sense that learners are required to 
engage in problem-solving activities. The more causal question of whether or not online PBL of 
this nature teaches or enhances problem-solving skills cannot be answered in an exploratory study 
and with this data. A future study examining the impact on student activities of both the 
instructional design of the course and the behaviour of the instructor might begin to shed some 
light on this question. 

Based on our results, it also appears that problem-solving in the online PBL context, as 
represented by Jonassen’s (1997, 2000) PFR process, is constrained by the instructional design of 
the course, and, therefore, somewhat limited in nature. Learners were not asked to engage in 
Articulating the Problem Space and only minimally focused on Defining the Problem Space, nor 
did they employ the process of Acting on Solutions. Jonassen (2000) differentiates between well- 
structured and ill-structured problems. Well-structured problems have a clear initial state (what is 
known) and the nature of the solution is well defined, while ill-defined problems have solutions 
that are not predictable or convergent which may also require the integration of several content 
domains. 

In this context, students were provided with limited information about the problems, but the 
problems also had well-defined initial states and required a clear and specific form of solution. 
Learners were presented with a clear problem statement (See Figure 1) and several further 
disclosures of information throughout the case. They were also presented on the second day with 
an extensive set of questions to consider, which helped the students considerably to determine the 
learning issues they had to address. In other words, they were well-structured problems. In order 
to more fully engage learners in the initial processes of the PFR model of problem solving, 
instructors and instructional designers may need to provide much less structure in terms of ill 
structured, open-ended problems and to allow for more flexibility in the directions that learners 
can take in arriving at solutions to the problems. It may, for instance, be advisable to avoid 
linking the PBL problems to specific marked assignments and, instead, base course assessments 
on other measures of the knowledge and skills acquired through the PBL process. 

Finally, while the course instructor regarded the problems as “quite realistic” and, therefore, 
having some of the characteristics of ill-structured problems by virtue of being similar to those 
are encountered in professional practice (K. Nolan, personal communication , July 15, 2005), they 
are not actual, real life problems in which the solutions are to be put into practice. PBL problems 
tied to field or practicum type experiences might well engender the full range of PFR behaviours 
and lead to a more complete problem-solving process. 

Barrows’ (2000) concept of “authentic” PBL is intended to support the application of problem- 
solving skills required in “clinical practice.” While this study provides some evidence that online 
PBL experiences can be designed which do foster problem-solving, more studies examining a 
range of online PBL contexts and instructional designs will be needed to confirm and detail this 
conclusion. 
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